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Abstract. Despite the fact that digital technologies are more and more used in the learning 
and education process, there is still lack of professional evaluation tools capable of assessing 
the quality of used digital teaching aids in a comprehensive and objective manner. Construc¬ 
tion of the Comprehensive Evaluation of Electronic Learning Tools and Educational Software 
(CEELTES) tool was preceded by several surveys and knowledge obtained in the course of cre¬ 
ation of digital learning and teaching aids and implementation thereof in the teaching process. 
The evaluation tool as such consists of sets (catalogues) of criteria divided into four separately 
assessed areas - the area of technical, technological and user attributes; the area of criteria 
evaluating the content, operation, information structuring and processing; the area of criteria 
evaluating the information processing in terms of learning, recognition, and education needs; 
and, finally, the area of criteria evaluating the psychological and pedagogical aspects of a digital 
product. The specified areas are assessed independently, separately, by a specialist in the given 
science discipline. The final evaluation of the assessed digital product objectifies (quantifies) 
the overall rate of appropriateness of inclusion of a particular digital teaching aid in the teach¬ 
ing process. 

Keywords: evaluation, educational software, electronic learning tools. 


1. Introduction 

Distinguishment of a high-quality educational product from other, less appropriate 
products, or even socially dangerous electronic didactic means, is an exceptionally 
difficult task. It requires determination of quality standards and existence of special 
evaluation tools respecting the specific features of electronic teaching and learning 
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aids and software applications with the ambition to educate (Karolcik et al., 2013). 
Despite the fact that there are relevant opinions which regard any strict limitations put 
to the educational software group as dangerous and inappropriate (Kalas, 2005; Lever- 
Duffy and McDonald, 2011; Foshat and Ahmed, 2003; Resnick, 2002), majority of 
the published studies accept the inclusion of a software product developed to support 
the education among special products which are subject to special requirements (Jeff, 
2013; Kalas, 2005; Lehotska, 2007; IT232 - Pedagogical software, 2011; Educational 
software, 2014). Unfortunately, production of multimedia learning materials is fre¬ 
quently not informed by relevant research in psychology and education. Therefore, 
the Internet offers a large number of completed digital educational objects of various 
quality levels (Leacock and Nesbit, 2007; Nesbit and Leacock, 2006; Shavinina and 
Loarer, 1999). 


2. Digital Teaching/Learning Aid Quality Assessment 

In times of general digitalisation of public and private life of the society, the objective 
and complex evaluation of digital teaching aid quality represents a great challenge for 
the current pedagogy and psychology. 

Attempts to identify key properties of software applications with regard to specific 
education requirements date back to the times of experimenting with software in schools 
and gradual integration thereof in the teaching agenda. At the turn of the millennium, 
evaluation tools were mostly in form of simple forms containing scaled items grouped 
into several categories (e.g. Content, Assessment, Technical Quality) (Kelly, 2002). 
They were intended for teachers. In their structured layout, teachers were evaluating 
basic characteristics of a particular used digital teaching aid and describing its strengths 
and weaknesses. Concluding parts of the form were focused on the summary of the ac¬ 
quired pedagogical experience and the description of teaching procedures and strategies, 
in which the use of a particular teaching aid brought good learning results. Exponential 
rise of the number of education-focused software applications and electronic teaching 
materials brought the need of categorising, grouping and dividing them. 

In an effort to mitigate the problem of searching the appropriate software, some 
information sources offer their own quality evaluation metrics, which is applied to a 
specific digital product and the measured values are then reflected in the search require¬ 
ments (Vargo et al., 2003). However, the efficacy of this technique of searching the 
quality teaching aids is directly dependent on applicability and appropriateness of the 
selected evaluation tools that generate the quality ratings. 

Evaluation of the quality of special teaching aids of the digital nature is discusses 
also in the part of the book named Evaluation and Selection of Learning Resources: A 
Guide (2008). Authors classified video files on DVDs (DVD/Video), digital software 
products (Digital Resources) and sources available on the web (Web Resources) as such 
teaching aids. Individual evaluation tools were grouped into four main categories: Con¬ 
tent, Educational Aspects, Technical Aspects, and Social Issues. They are further di¬ 
vided into individual criteria groups and subgroups. As for the digital teaching aids, the 
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evaluation criteria groups in categories of education and technical aspects are extended 
with additonal assessed characteristics, such as simple navigation in the program envi¬ 
ronment (User navigation through program is appropriate), interaction-supported mean¬ 
ingful learning (Interaction promotes meaningful learning), possibility to integrate the 
program in other teaching activities within the subject (The program can be integrated 
with other activities in the same subject), immediate, positive and motivational nature 
of the feedback (Feedback is non-threatening, immediate, positive, motivational, and 
user-sensitive), and as for technical aspects, for example availability of the help function 
(A help function is provided and appropriate), or the possibility for a teacher to control 
a student’s progress and achieved results (Teacher control of feedback and progress is 
provided and appropriate). Annexes to the materials also include three separate evalu¬ 
ation forms by Kathleen Schrock, particularly created for the evaluation of quality of a 
website with the educational content (Critical evaluation of a web site) from the teach¬ 
er’s point of view and from the point of view of educational needs in the first and second 
levels of basic schools. (For use by educators, Elementary school level and Secondary 
school level). 

Criteria catalogues were developed for purposes of evaluating educational software 
products in the German-speaking environment. The authors of this study among other 
things stress the fact that the process of the evaluation of educational software is often 
marked by subjectivity and that is why the purpose of the evaluation of the software 
(Learnsoftware) is not its inclusion into the category good or inappropriate. They also 
stress the importance of the following function of the existence of the catalogue of eval¬ 
uation criteria which should improve creation of new educational software mainly from 
the pedagogical point of view (Gottfried et al., 2002). 

The thought-out and well-elaborated methodology of the digital learning object eval¬ 
uation (The Learning Object Review Instrument - LORI) is presented by the study by 
authors Leacock and Nesbit (2007). Digital learning objects are small, simple, modular 
and discrete units designed especially for electronic delivery and use (What are Digital 
Learning Objects?, 2014). Authors of the study (Leacock and Nesbit, 2007) realize that 
the described learning object review instrument (LORI) is not sufficient tool for the 
comprehensive evaluation of the quality of educational programs consisting of a large 
number of individual learning objects. 

Evaluation of multimedia, pedagogical and interactive software in the context of 
the teaching and education process represents the topic of the study by French authors 
(Crozat, S., Hu, O., Trigano, R) which characterises in more details the EMPI method 
(Evaluation of Multimedia, Pedagogical and Interactive software) (Crozat et al., 1999; 
Crozat et al., 1999). They point out the fact that the general precondition of elaboration 
of a usable evaluation method is to know the answer to 3 important questions: Who is 
the evaluator, what is the subject of the evaluation, and when it is evaluated (Depover, 
1994). In case the EMPI method is used, the evaluators will be the users of multimedia 
software, i.e. experts in pedagogy, didactics, methodologists, or instructors working in 
education centres. The evaluation only applies to finished software products and the 
proposed evaluation system divides the evaluation criteria into 6 separate modules. For 
each of them, special evaluating forms were created. 
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3. Methodology for the Creation of the Evaluation Tool Concept 

Elaboration of the concept for the comprehensive evaluation of educational software and 
electronic learning materials was preceded by the performance of two survey projects. 
The methodology applied within the initial survey was based on the Delfi method. In 
2013, questionnaires with a special structure, containing exclusively open items, were 
administered to 3 groups of addressed experts who regularly use digital technologies in 
their usual working or private life. These experts provided their statements regarding 3 
issue areas related to the expected most important properties of software applications 
and electronic materials especially designed for the education needs and teaching sup¬ 
port. The survey brought rather definite results and the conclusion that the simplicity and 
lucidity of installations, launch and orientation belong to the most significant properties 
of education-based software products (Karolcik et al., 2013). The second survey was 
carried out with the sample of 1,842 teachers - graduates of specialized education in the 
field of efficient and reasonable use of digital technologies in the educational process 
that was carried out within the National Project of Modernization of Educational Process 
at Elementary and Secondary Schools (MVP, 2014). All the addressed respondents suc¬ 
cessfully passed their courses by the elaboration of the final thesis, defence thereof, and 
the final exam. Thereby, they proved an adequate level of knowledge about a suitable 
and methodically well-prepared application of digital technologies in the teaching pro¬ 
cess. They can thus be regarded as experts in this area and their opinions on the issues of 
digital learning aids quality evaluation as substantial and important. The used question¬ 
naire with a specialised structure contained closed questions with multiple answer op¬ 
tions and scaling items. The survey was exploring the opinions and attitudes of teachers 
regarding the issues of significance of digital learning aids quality evaluation by inde¬ 
pendent experts and the optimal method of publishing the achieved results, including the 
methodology of the appropriate use of the aids in education. The questions with multiple 
answer options inquired about the preferences among teachers with regard to the most 
important properties of software products with the ambition to educate. The research 
confirmed, inter alia, that teachers regard as exceptionally important the knowledge of 
the quality of available digital teaching aids and such information, supplemented with 
suggestions, methodologies, and recommendations for didactically suitable use of a par¬ 
ticular aid, would be very helpful for them in their work. However, they expect that the 
evaluation will be elaborated by experts - specialists with required qualification, who 
perfectly understand the issues (fields of science) in which they evaluate a product. The 
summary results from the questionnaire survey, including the detailed statistical analysis 
of the measured data, will be processed in a separate scientific study. 

Our concept of the Comprehensive Evaluation of Electronic Learning Tools and Ed¬ 
ucational Software (CEELTES) uses the knowledge of the existing evaluation tools and 
published experience with their use or development. It considers important conclusions 
and observations from our research. Unlike majority of known evaluation concepts, it 
minimises the rate of subjective freedom of an evaluator when assessing the determined 
criteria and maps more precisely (in more details) all the specified key characteristics 
of a particular digital teaching aid. The risk connected with incorrect or imprecise un- 
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derstanding of the content of the specified criteria by the evaluator is minimised here by 
the choice of experts - specialists in the given field of science. Each of them evaluates a 
digital teaching aid always with regard to a single evaluation topic. 

Extraordinary width and variety of software products usable in the teaching process 
complicates, to a large extent, the implementation of unified and simple patterns for 
their evaluation. This problem is solved by authors of evaluation tools by elaboration 
of several questionnaires which always evaluate a single type of a digital teaching 
aid. Our concept of the Comprehensive Evaluation of Electronic Learning Tools and 
Educational Software (CEELTES) offers a simple model of the evaluation methodol¬ 
ogy. An evaluator selectively chooses from the range of criteria only those, for which 
their applicability or inapplicability can be assessed (assumed) for a particular type of 
a digital product. 

As it is not possible to use the LORI as a general instrument for the evaluation of 
digital learning aids and the EMPI method does not take into account extraordinary 
diversity and variety of available technological solutions, we based our proposal of 
the evaluation areas definitions and specification of individual criteria on the group of 
complex learning tools. These learning tools represent the most complex, very thor¬ 
oughly prepared and elaborated software solutions designed especially for the teaching 
and learning purposes. They mostly integrate also other groups of products, such as 
instructing programs, applets, special software modules or complements, into a single 
functional unit. Particularly an extraordinary wide range of the offered educational con¬ 
tent, a wide range of various functions, settings, and special instruments for the system 
administration, enable the assessment of the largest number of product quality aspects 
(Karolcik et ah, 2013). 


4. The Comprehensive Evaluation of Electronic Learning Tools and 
Educational Software (CEELTES) 

The elaboration of the concept for the comprehensive evaluation of the quality of educa¬ 
tional software and electronic learning materials was based on five key requirements: 

• Educational software (complex educational environment, applet, software module 
or a complement) should be stable and provide the users with a high-level techni¬ 
cal, technological and user comfort. 

• Presented digital content should provide a thorough, age-appropriate, method¬ 
ologically reasonable, high-quality, and detailed view on the given topic, consid¬ 
ering the pedagogical and psychological aspects of learning, respecting the cur¬ 
ricula and using the latest information processing technologies. 

• The use of educational software should be simple and intuitive. User graphic in¬ 
terface should be comprehensible and the selected software operation tools should 
be clear, understandable and age-appropriate. 

• Digital content should be open, offer various interactive educational activities 
supporting the learning, recognition and exploration, provide the feedback, and in 
the widest possible extent use visualization and multimedia. 
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• Educational software (complex educational environment, applet, software module 
or a complement) must be supplemented with various supporting didactic materi¬ 
als, work sheets, handbooks on how to work with its digital content properly (in an 
appropriate and suitable manner) and use it reasonably in the education process. 

The key requirements were divided into four separate areas: 

1. The area of technical and technological attributes. 

2. Content, operation, information structuring and processing. 

3. Information processing in terms of learning, recognition and education needs. 

4. Psychological and pedagogical aspects. 

The evaluation areas consist of the sets (catalogues) of evaluation criteria and result 
quantification proposals. Each evaluation area is assessed by a qualified expert -a spe¬ 
cialist in the given area. This minimises the incorrect or inadequate understanding of the 
content of the determined criteria. Practical use of the proposed point scale is complicated 
by a high diversity of the existing software products. Therefore, quality assessment with 
a larger group of criteria within the assessed area requires also a summary evaluation of 
the measured properties of electronic learning aids. For the same reason, an evaluator 
does not have to use all the defined criteria and can choose only those, applicability or 
delivery of which can be objectively assessed in a particular digital product. Criteria, for 
which the point assessment is not possible, are not included in the overall area evalua¬ 
tion. Every determined area is evaluated separately and autonomously without identify¬ 
ing a precise proportion of possible participation in the overall evaluation result. 

The summarising and complete list of assessed criteria included in the concept of 
comprehensive evaluation of the quality of educational software and electronic learning 
material is presented in separate chapters. It is also included in the electronic content of 
the website http://www.evaluedu.sk/sk/home/evaluation-of-elec- 
tronic-learning-tools-and-educational-software/. The article pre¬ 
sents only a fundamental structure of the comprehensive evaluation instrument and 
examples of the criteria in selected evaluation areas. Maximum number of points assign¬ 
able to an assessed learning aid in a particular criteria area is shown in brackets. 

1. Technical and technological attributes 

Any high-quality, methodically well prepared and excellently processed electronic 
content of a digital teaching aid ceases to be educationally attractive, if technical prob¬ 
lems disable to use it in the teaching process. Technical and technological parameters of 
a digital teaching aid are not directly related to the quality of processing of the presented 
educational content. However, they significantly affect the extent of possible application 
of the aid in the educational practice. Moreover, structured information about technical 
and technological attributes of a digital teaching aid has also an informative function, 
as it notifies a user about the limitations of a selected technological platform and the 
requirements for the technical equipments in schools. 

Defined groups of evaluation criteria: 

l.a Installation and launch (23 points) 

• Application acquisition does not require formal registration of a user 
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• Application does not require the Internet connection and its functionality is not 
conditioned by a high speed of the Internet connection (transmission efficiency). 

• Launch is smooth, fast, standard licence conditions are confirmed only at the first 
launch. 

• Launch is not conditioned by (unnecessary/redundant) user identity verification. 

• Software is a web application 

• Application does not require installation of special plugins intended for specific 
web browsers. 

• Application runs correctly and without any problems in most frequently used 
web browsers (e.g. Microsoft Internet Explorer, Google Chrome, Mozilla Fire- 
fox, Opera). 

• Application explicitly does not require installation of the latest version of a web 
browser. 

• Software is not a web application 

• Application does not require installation of any additional software (except for 
the application’s installation file). 

• Installation runs smoothly, instructions for users are comprehensible, clear and 
continuously inform users about the installation progress for individual seg¬ 
ment. 

• Application can be shared in a network (Intranet), or there is a network version 
thereof. 

• Application is launched directly from the medium, not requiring the assigned 
administrator rights. 

• Application is launched directly from the medium and the launch is not condi¬ 
tioned by entering any special code with each new use, or access to the HW key. 

• Detailed definitions were determined also for the groups of criteria evaluating the 
technical, technological, and user attributes in the following areas: 

Lb Stability (10 points) 

l.c Support (12 points) 

l.d Hardware (6 points) 

l.e Operating system (5 points) 

l.f Used technologies {5 points) 

1. g Safety (4 points) 

2. Content, operation, information structuring and processing 

Due to its nature and focus, the second group of the evaluation criteria is partially 
inspired by the key requirements related to textbooks and educational texts. This group 
is supplemented with the lists of criteria that evaluate technical and technological limits 
of a selected program platform used in the presentation of the educational content. 
Defined groups of evaluation criteria: 

2. a Content (50 points) 

• Software content respects the fundamental pedagogical documents - national cur¬ 
riculum, educational programs, content and performance standards. 
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• Information processing by the software develops the key competences of a learner 
prescribed in the applicable and binding pedagogical documents and the informa¬ 
tion is presented in a logical structure. 

• Software content is completely localised into the particular mother tongue (na¬ 
tional language). 

• Software content provides the themes for educational projects, problem tasks and 
real-life examples. 

• Presented information can be used in an adjusted form in separate presentations. 
Texts are stylistically and professionally adjusted to the age of users in the target 
group and develop their communication skills. 

• Software content provides the knowledge, facts, and definitions which are profes¬ 
sionally and objectively correct, obtained from trustworthy sources, and verified. 

• Information provided by the software is up-to-date, accurate, in various graphic 
formats - tables, graphs, models, etc. 

• Information texts are supplement by additional explanatory information and other 
information sources, in an appropriate manner. 

• Software contains high-quality, varied, and updated picture material, considering 
the specific features of the given subject and the key competencies developed 
thereby. 

• Processing of the information by the software facilitates good text readability. 

• Important terms are highlighted, indicated in a different colour, and the text is 
divided into paragraphs. 

• Used picture and film material is pertinent, easily identifiable, and comprehen¬ 
sible, corresponding to the text information. 

• Quality of the picture and film material used in the software is high, with clear 
colours, developing aesthetic feeling in users. 

• Software content meets the social correctness requirements, is processed in an 
objective manner and in compliance with the respect for human rights and free¬ 
doms. 

• Detailed definitions were determined also for the groups of criteria evaluating 
the content, operation, information structuring and processing in the following 
areas: 

2,b Operation (16 points) 

2.c Information structuring and processing (14 points) 

3. Information processing in terms of learning, recognition, and education needs 

If we regard didactic means to be any sources of material, as well as non-material 
nature, which facilitate the fulfilment of the determined educational objectives, the edu¬ 
cational software product represents one of the most up-to-date forms thereof. There¬ 
fore, it is particularly important that a digital teaching aid serves also as motivation for 
users, provides them with quality feedback, and enables them, in a required extent, to 
control their learning or teaching. Authors of a digital teaching aid must also provide 
a high-quality professional methodical assistance and didactic support. This purpose 
would be best served by practical examples of teaching activities, including procedures 
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and suggestions how to integrate, in a proper and meaningful manner, the educational 
software in the teaching agenda. Our research confirmed that particularly the examples 
how to use a digital teaching aid are regarded by teachers as the most usable in their 
pedagogical practice. 

Defined groups of evaluation criteria: 

3.a Didactic information processing (24 points) 

• Thanks to the diversity and variety of the content, the software enables a teacher 
to use a wide range of teaching methods and procedures. 

• Software is open and assumes the provided information database will be subse¬ 
quently used in the learning process and does not hinder that. 

• Software can be used in all stages (sections) of a lesson. 

• Software contains the functions enabling (simplifying) the printing, storage, shar¬ 
ing, or export of the presented information. 

• Information texts are linguistically correct, stylistically adequate, used vocabulary 
corresponds to the age of users in the target group. 

• Information texts are written in an interesting and comprehensible manner, sen¬ 
tence length and text segmentation into paragraphs corresponds to the age of users 
in the target group. 

• Hypertext links refer to other information sources and the related facts are high¬ 
lighted in the text block and easy to distinguish. 

• Detailed definitions were determined also for the groups of criteria evaluating the 
information processing in terms of learning, recognition and education needs in 
the following areas: 

3.b Motivation and activation of learning activities (20 points) 

3. c Methodical support (16 points) 

4. Psychological and pedagogical aspects 

Defined groups of evaluation criteria: 

4. a Current level of cognitive development of learners and stimulation of 

their cognitive development (24 points) 

For the learners in the secondary stage of elementary schools they provide the fol¬ 
lowing: 

• Opportunities to consider possible, hypothetical, and unreal situations and pos¬ 
sibilities and also the consequences of their update (solution of tasks with unreal 
input data). 

• Complex tasks requiring systematic solution procedures, search through the pos¬ 
sibilities, creation of strategies, probability-based thinking. 

• Mediated information requires understanding, interpreting of, and thinking about 
abstract terms, ideas, and relations. 

• Opportunities to search and formulate abstract relations (write an equation, ex¬ 
press the relation between the variables and the constants in a simple proportion, 
construct the formula for the calculation of properties of certain objects, analyze 
various mathematical, physical, chemical, economical, and other formulas, draw 
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the structure of the problem situation, alternative strategies of the problem solv¬ 
ing, etc.), use them when considering and solving problems. 

4.b Activity level in the educational activities (15 points) 

• Educational activities focused on a meaningful, logical organizing of terms, ideas, 
and visual components mediated by the learning software (active information 
processing, structuring it into a comprehensive understanding of the acquired 
contents, which is demonstrated by deducting meaningful, logical relations, ap¬ 
plications, etc.), asking verification questions to verify the level of meaningful 
learning. 

• Learning activities focused on presenting examples of acquired terms, related de¬ 
tails, relations to other terms and concepts that were the subject of previous learn¬ 
ing, creation of more complex interdisciplinary relations and applications of the 
curriculum content in everyday life conditions, verification questions and tasks to 
verify the aforesaid relations and applications. 

• Requirements for active information processing and interpretation, using the 
previous knowledge and thinking (e.g. formalizing the structures of one’s own 
knowledge in form of term maps, thinking about consequences of manipulations, 
changes in the environment on the level of thought experiment, manipulations in 
the virtual environment, etc.). 

• Possibility to solve tasks and problems as the source of new knowledge, i.e. ap¬ 
plication of learning on the individual or the group levels using the cooperative 
learning. 

• Detailed definitions were determined also for the groups of criteria evaluating the 
psychological and pedagogical aspects in the following areas: 

4.c Constructiveness in learning activities (10 points) 

4.d Intentionality in learning activities (10 points) 

4.e Authenticity in learning activities (10 points) 

4.f Taking into account the social nature in learning activities (10 
points) 

4.g Taking into account the requirements following from the behavioural 
learning theory (26 points) 

4.h Taking into account the requirements following from the cognitive 
theory (20 points) 

4.1 Taking into account individual differences (general and academic 
intelligence) (15 points) 

4.j Taking into account individual differences (learning and cognitive 
styles) (10 points) 

4.k Taking into account individual differences (profile of multiple intel¬ 
ligences) (5 points) 

4.1 Taking into account individual differences (types of preferred learn¬ 
ing motivation) (5 points) 

Overall evaluation of a digital learning aid 

The final, complex evaluation of the assessed educational software (electronic learning 
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aid) includes the number of assigned points in individual evaluation areas (rough score) 
and the percentage representing the ratio to the maximum possible number of the as¬ 
signed points. These indicators are supplemented with the verbal evaluation divided into 
three categories. Verbal evaluation degrees take into account the fact that we work with 
a strict model of evaluating the quality of educational digital product that compares the 
existing software products with the ideal model. Therefore, exceeding 50% of the point 
score limit in 3 evaluation areas can be regarded as a very good result and the evaluated 
digital learning aid as a very good tool. An excellent digital learning aid is characterised 
by at least 80% of the assigned points in 3 out of 4 evaluated areas and good aid in case 
of more than 30%. A software product that failed to obtain at least 30% of the total num¬ 
ber of points in majority of the evaluation areas, as for the quality evaluation, cannot be, 
in our opinion, regarded as educational, even though its individual use does not exclude 
interesting educational results. 


5. Conclusion 

The Comprehensive Evaluation of Electronic Learning Tools and Educational Soft¬ 
ware (CEELTES) represents a special and modular evaluation instrument for the as¬ 
sessment of digital learning aid quality. The system consists of 4 evaluation areas 
consisting of a set (catalogue) of evaluation criteria and the interval quantification of 
the achieved results. The preparation and development of the system was preceded by 
several research projects. They mapped the extent and nature of the available software 
solutions, as well as the properties expected from the software applications and elec¬ 
tronic materials especially intended and designed to support learning by the users of 
digital technologies. 
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Annexes 


The Comprehensive Evaluation of Electronic Learning Tools and 
Educational Software (CEELTES) 


Area - Technical and technological characteristics 


Evaluation criteria * Maximum 

(* If a digital learning aid meets the criteria in the given sub-area, it is assigned number of 

the maximum number of points shown in brackets! In case it meets the criteria points 
only partially, the score is reduced accordingly. If the evaluator is not able to assess 
objectively the applicability or inapplicability of any of the determined criteria, such 
criterion is skipped and the maximum number of points shown in the brackets next 
to the criterion will not be included in the overall score.) 

Abbreviations used for the assessed platforms: PC (computer/laptop), MOB (mobile/ 
tablet), WEB (web application) 


La 

Installation 
and launch 

(applica¬ 
tion acqui¬ 
sition, ins¬ 
tallation 
and 
launch) 


Application acquisition does not require formal registration of a user 
( 0-3 points) 

Application does not require the Internet connection and its functionality is not 
conditioned by a high speed of the Internet comiection (transmission efficiency) 
( 0-3 points) 

Launch is smooth, fast, standard licence conditions are confirmed only at the 
first launch 
( 0-2 points) 

Launch is not conditioned by (unnecessary/redundant) user identity verification 
( 0-2 points) 


23 


[WEB] Software is a web application (13 points) 

■ Application does not require installation of special plugins intended for specific 
web browsers 

(0-4 points) 

■ Application runs correctly and without any problems in most frequently used web 
browsers (e.g. Microsoft Internet Explorer, Google Chrome, Mozilla Firefox, 
Opera) 

(0-6 points) 

■ Application explicitly does not require installation of the latest version of a web 
browser 

(0-3 points) 


Lb 

Stability 


[PC, MOB] Software is not a web application (13 points) 

■ Application does not require installation of any additional software (except for 
the application’s installation file) 

(0-4 points) 

■ Installation runs smoothly, instructions for users are comprehensible, clear and 
continuously inform users about the installation progress for individual segment 
(0-2 points) 

■ [PC] Application can be shared in a network (Intranet), or there is a network 
version thereof 

■ [PC] Application is launched directly from the medium, not requiring the 
assigned administrator rights 

(0-2 points) 

■ [PC] Application is launched directly from the medium and the launch is not 
conditioned by entering any special code with each new use, or access to the 
HW key 

(0-2 points) 

■ Application works correctly, without any problems; repeated launch is not 
required in case of short and temporary Internet connection failure (if required 
by the application) 

(0-2 points) 
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l.c 

Support 


l.d 

Hardware 


l.e 

Operating 

system 

1-f 

Used tech¬ 
nologies 

1-8 

Safety 


■ Application works correctly, without any problems; repeated launch is not 
required in case of short and temporary Internet connection failure (if required 
by the application) 

(0-2 points) 

■ Application works correctly, without any problems, repeated launch is not 
required in case of accidental, but also intentional, undesired (not recommended) 
interventions by a user 

(0-4 points) 

■ Application does not freeze, crush, does not require repeated installation, and 
does not show any failures with error reports 

(0-4 points) 

■ Application is regularly updated, improved - upgrade 
(0-3 points) 

■ Application has a permanent technical support (e-mail, hotline, instant messaging, 
etc.) 

(0-3 points) 

■ Application has a special website containing the basic information and recom¬ 
mendations 

(0-2 points) 

■ Instruction manual is available for the application 
(0-4 points) 

■ Application does not require any extraordinary system requirements for HW 
(0-2 points) 

■ Application does not impose any high, above-standard hardware requirements 
(e.g. RAM, HDD, processor, graphics card performance, ...) 

(0-4 points) 

■ Application exists for various operating systems - MS Windows, Linux, Mac 
OS, Android, iOS, Windows Mobile... 

(0-5 points) 

■ Software uses modem information processing and presentation technologies 
(0-5 points) 


■ If the application stores/transfers user data 

■ [WEB] Application runs under a secure protocol (e.g. HTTPS) 

(0-4 points) 

■ [PC, MOB] In case the application stores the data in files, they are adequately 
secured against misuse 

(0-4 points) 

Altogether for area 1 


12 


6 


5 


5 


65 


Area - Content, operation, information structuring and processing 


Evaluation criteria * 

(* If a digital learning aid meets the criteria in the given sub-area, it is assigned 
the maximum number of points shown in brackets! In case it meets the criteria 
only partially, the score is adequately reduced. If the evaluator is not able to assess 
objectively the applicability or inapplicability of any of the determined criteria, such 
criterion is skipped and the respective maximum number of points shown in the 
brackets will not be included in the overall score.) 

2.a 2.a,l 50 

Content ■ content respects the fundamental pedagogical documents - national 

curriculum, educational programs, content and performance standards. 

(0-4 points) 

■ Information processing by the software develops the key competences of a 
learner prescribed in the applicable and binding pedagogical documents and the 
information is presented in a logical structure 
(0-2 points) 


Maximum 
number of 
points 
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■ Software content is completely localised into the particular mother tongue 
(national language). 

( 0—6 points) 

■ Software content provides the themes for educational projects, problem tasks and 
real-life examples. 

(0-2 points) 

■ Presented information can be used in an adjusted form in separate presentations. 
(0-2 points) 

■ Texts are stylistically and professionally adjusted to the age of users in the target 
group and develop their communication skills. 

(0—2 points ) 

2.a,2 

■ Software content provides the knowledge, facts, and definitions which are profes¬ 
sionally and objectively correct, obtained from trustworthy sources, and verified 
(0—4 points ) 

■ Information provided by the software is up-to-date, accurate, in various graphic 
formats - tables, graphs, models, etc. 

(0—4 points ) 

■ Information texts are supplement by additional explanatory information and other 
information sources, in an appropriate manner 

(0-2 points) 

■ Software contains high-quality, varied, and updated picture material, considering 
the specific features of the given subject and the key competencies developed 
thereby 

(0—6 points) 

2. a. 3 

■ Graphic processing of the information by the software facilitates good text 
readability 

(0—2 points) 

■ Important terms are highlighted, indicated in a different colour, and the text is 
divided into paragraphs 

(0—4 points) 

■ Used picture and film material is pertinent, easily identifiable, and comprehensible, 
corresponding to the text information 

(0—4 points) 

■ Quality of the picture and film material used in the software is high, with clear 
colours, developing aesthetic feeling in users 

(0—2 points) 

2.0.4 

■ Software content meets the social correctness requirements, is processed in 
an objective manner and in compliance with the respect for human rights and 
freedoms 

(0—4 points) 


2.h 

Operation 


2.b.l 

■ Software operation is simple and intuitive 
(0—2 points) 

■ Graphical tools for the software operation are comprehensible, functionally ade¬ 
quate, and explicit 

(0-2 points) 

■ Software operation tools facilitate fast movement in the information database 
(0—2 points) 

■ Software enables users to monitor accurately their operations inside the software 
structure, repeat them, or it can remember a direct path to it (saves it in the 
memory) 

(0—2 points) 

■ Software operation tools include the functions of fast search, return to the start, 
and step back 

(0—2 points) 
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■ Software does not respond to a user’s steps related to the lack of attendance, 
accidental motion, or which are aimed at damaging its functionality 

(i 0-2 points ) 

■ Software operation does not hinder the work with other program applications and 
the Internet search 

( 0-2 points) 

■ (Software focused on science branches focused on the Earth) Software enables 
a user to move over the interactive map, where the marked locations are 
interconnected with the corresponding information database 

( 0-2 points) 


2.c 

Informa¬ 
tion struc¬ 
turing and 
processing 


2.c.l 

■ Software structure is well-organised, logical, and thoroughly thought-out. 
Orientation inside the structure is easy for users 

(0-2 points) 

■ Information is arranged in blocks, graphically clearly divided, segmented into 
separate thematic units 

(0-2 points) 

■ Software contains majority of well-known information formats - photographs, 
pictures, animations, audio and video recordings,... 

(0-4 points) 

■ All the pieces of information are mutually interconnected by means of hypertext 
links 

(0-2 points) 

■ Software works in the environment of independently controlled floating windows 
that can be individually positioned, activated and hidden 

(0-2 points) 

■ Software thoroughly, cartographically, and accurately describes spatial relations 
among the objects in the land area 

(0-2 points) 


Altogether for the area 2 


14 


80 


Area - Information processing in terms of learning, recognition, and education 
needs 


Evaluation criteria * Maximum 

(* If a digital learning aid meets the criteria in the given sub-area, it is assigned number of 
the maximum number of points shown in brackets! In case it meets the criteria points 
only partially, the score is adequately reduced. If the evaluator is not able to assess 
objectively the applicability or inapplicability of any of the determined criteria, such 
criterion is skipped and the respective maximum number of points shown in the 
brackets will not be included in the overall score.) 


3. a 

Didactic 
informati¬ 
on proces¬ 
sing 


3.a.l 

■ Thanks to the diversity and variety of the content, the software enables a teacher 
to use a wide range of teaching methods and procedures 

(0-4 points) 

■ Software is open and assumes the provided infonnation database will be 
subsequently used in the learning process and does not hinder that 

(0-4 points) 

■ Software can be used in all stages (sections) of a lesson 
(0—4 points) 

■ Software contains the functions enabling (simplifying) the printing, storage, 
sharing, or export of the presented information 

(0-6 points) 

3.a.2 


■ Information texts are linguistically correct, stylistically adequate, used vocabulary 
corresponds to the age of users in the target group 

(0-2 points) 

■ Infonnation texts are written in an interesting and comprehensible manner, 
sentence length and text segmentation into paragraphs corresponds to the age of 
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users in the target group 
(0-2 points) 

■ Hypertext links refer to other information sources and the related facts are 
highlighted in the text block and easy to distinguish 
(0-2 points) 


3.h 

Motivation 

and 

activation 
of learn ing 
activities 


3.b.l 

■ Software contains interactive educational tasks (tests) and continuously or 
immediately evaluates the success rate of answers 

(0-6 points) 

■ Software contains interactive educational games, motivation and entertainment 
activities related to the given topic 

(0-6 points) 

■ Software contains themes for educational projects, problem tasks, work sheets 
in a digital form, and a handbook on how to efficiently work with the software 
at the lessons 

(0-8 points) 

■ (Testing software ) Software monitors and evaluates in graphs and tables the 
continual success rate in the testing of several users 

(0-4 points) 

■ (Testing software ) Software provides the success chart of individuals within the 
group, informs about their progress in time and in certain testing stages 

(0-4 points) 

■ (Testing software) Individual testing items (questions, tasks) in the interactive 
tests are explicit and use various multimedia formats 

(0-6 points) 

■ (Testing software) Software enables a teacher to lead, control, and organise the 
testing 

(0-2 points) 

■ (Testing software) Software enables to choose tasks from the database and produce 
specific tests based on a user’s requirements 

(0-2 points) 

■ (Testing software) Software enables printing out of all the produced tests and 
administer them in a conventional manner (in writing) 

(0-2 points) 


3.c 3.c.l 

Methodical ■ s 0 ft ware j s supplemented with a printed or electronic handbook, user manual, or 
support a video, a course with tips, ideas on how to use and work with the software 

(0-4 points) 

■ Software has a special website containing the information and the methodical 
recommendations how to work with the software at the lesson, or in which 
education stage the use thereof brings the best educational effect and results 
(0-4 points) 

■ Software website is regularly updated and supplemented with new didactic 
materials, work sheets, tasks, and practical experience of users with the software 
application at the lessons 

(0-6 points) 

■ Teachers can actively participate in the production of methodology, projects, 
educational activities contained in the software website 

(0-2 points) 


Altogether for area 3 


20 


16 


60 


Area - Psychological and pedagogical aspects 

Evaluation criteria * Maximum 

(* If a digital learning aid meets the criteria in the given sub-area, it is assigned number of 
the maximum number of points shown in brackets! In case it meets the criteria points 
only partially, the score is adequately reduced. If the evaluator is not able to assess 
objectively the applicability or inapplicability of any of the determined criteria, such 
criterion is skipped and the respective maximum number of points shown in the 
brackets will not be included in the overall score.) 
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4.a Taking into account the current level of cognitive development of learners and stimulation of their 
cognitive development 

4.a.l (ISCED 1) 24 

■ Examples of terms, processes, and relations are presented in a demonstrative, illustrative, 
figurative form (e.g. pictures, demonstrative schemes, realistic video sequences, simulations, 
animations, etc.) 

( 0-8 points) 

■ Tasks including data processing and results presentation in a graphical form, (0-5 points) 

■ Tasks focused on learning new terms and their mutual relations based on analyses, comparisons, 
classification, and abstraction of term properties 

(0-7 points) 

■ Simple attempts or experiments in the virtual environment containing the demonstrative 
components 

(0-4 points) 

4.0.2 (ISCED 2 and ISCED 3a) 

■ Opportunities to consider possible, hypothetical, and real situations and possibilities and also 
the consequences of their update (solution of tasks with unreal input data) 

(0-8 points) 

■ Complex tasks requiring systematic solution procedures, search through the possibilities, 
creation of strategies, probability-based thinking ... 

(0-5 points) 

■ Mediation of information requiring understanding, interpreting of, and thinking about abstract 
terms, ideas, and relations 

(0-4 points) 

■ Opportunities to search and formulate abstract relations (write an equation, express the relation 
between the variables and the constants in a simple proportion, construct the formula for the 
calculation of properties of certain objects, analyze various mathematical, physical, chemical, 
economical, and other formulas, draw the structure of the problem situation, alternative 
strategies of the problem solving, etc.), use them when considering and solving the problem 
(0-7 points) 

4.b Taking into account the activity level in the learning activities 

■ Learning activities focused on a meaningful, logical organizing of terms, ideas, and visual 15 

components mediated by the learning software (active information processing, structuring it into 

a comprehensive understanding of the acquired contents, which is demonstrated by deducting 
meaningful, logical relations, applications, etc.), asking verification questions to verify the level 
of meaningful learning 
(0-4 points) 

■ Learning activities focused on presenting examples of acquired terms, related details, relations 
to other terms and concepts that were the subject of previous learning 

(0-3 points) 

■ Creation of more complex interdisciplinary relations and applications of the curriculum content 
in everyday life conditions, verification questions and tasks to verify the aforesaid relations and 
applications 

(0-2 points) 

■ Requirements for active information processing and interpretation, using the previous 
knowledge and thinking (e.g. formalizing the structures of one’s own knowledge in form of 
term maps, thinking about consequences of manipulations, changes in the environment on the 
level of thought experiment, manipulations in the virtual environment, etc.) 

(0-3 points) 

■ Possibility to solve tasks and problems as the source of new knowledge, i.e. application of 
learning on the individual or the group levels using the cooperative learning 

(0—3 points) 

4.c Taking into account the constructiveness in learning activities 

■ Learners can verify the degree of curriculum content acquisition in terms of understanding it in 
relation to the previous learning and knowledge (e.g. in form of tasks to supplement the term 
map with other components and describe the relations between the components, solve the tasks 
to interpret the relations between the terms, concepts, ideas, formulas, etc. (0—3 points) 
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■ The feedback is sufficiently frequent, in terms of understanding the curriculum content it is 10 

instructive and induces higher cognitive processes of learners (applying creativity, critical 

thinking when considering and solving a problem, self-regulation - possibility to verify the 
level of one’s own knowledge, setting the learning objectives, selection of learning procedures, 
etc.) 

( 0-3 points) 

■ Possibility to solve tasks and problems that reveal basic explanatory concepts and mechanisms 
of the curriculum content 

( 0-2 points) 

■ Learners’ possibility to intensify their understanding of the curriculum content by active 
cognitive activities with regard to their own needs and interests (applying the gained knowledge 
and skills in other cognitive and practical contexts) 

( 0-2 points) 

4.d Taking into account the intentionality in learning activities 

■ Learners can select their own learning objectives 10 

( 0-3 points) 

■ They can choose tasks and problems, management of which relates to the learning objectives 
selected by them and takes into account their aspiration level corresponding to the task 
difficulty 

( 0-3 points) 

■ Learners can express and satisfy their cognitive needs and interests activated in them during the 
study (extend, intensify their knowledge and skills in other areas or activities) 

( 0-4 points) 

4.e Taking into account the authenticity in learning activities 

■ Learners acquire the curriculum content in sufficiently elaborated connections with reality and 10 
its practical application 

( 0-3 points) 

■ Learners can solve tasks and problems in real and practical contexts 
( 0-4 points) 

■ Learners acquire not only the learning contents, procedures, but also their sources, either by 
their own exploration or mediated via references to work of scientists or explorers 

( 0-3 points) 

4f Taking into account the social nature in learning activities 

■ Possibility for learners to communicate, cooperate in the process of acquisition of the mediated 10 
meaningful information 

( 0-5 points) 

■ Possibility for learners to cooperate in solving tasks, problems, or projects, including learning 
through exploration 

( 0-5 points) 

4.g Taking into account the requirements following from the behavioural learning theory 

■ Providing the continuous (immediate) and final feedback that informs the learners about the 
correctness or faultiness of their solutions 

( 0-3 points) 

■ The feedback includes instructions informing the learners on the subsequent procedure and are 
of the motivation value for them 

( 0-2 points) 

■ The program enables the learners make corrections and evaluate 
( 0-2 points) 

■ Opportunities for learners to attend to learning and game-based educational activities which are 
attractive for them, after they manage the compulsory learning activities 

( 0-2 points) 

■ While solving more complex tasks, in irregular intervals (depending also on a learner’s previous 
success) the software evaluates the correctness of the procedure; in case the procedure or the 
solution is correct, it provides substantiation; in case it is not correct, it provides a learner with 
the guidance for further learning or a procedure 

( 0-3 points) 
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■ An alternative in solving more complex tasks (objectives) is their segmentation in partial 20 

tasks (objectives); an optimal situation is when the segmentation takes into account individual 
differences in learners’ knowledge, skills, and abilities; work results corresponding to the 
objectives and tasks are then evaluated by the software for correctness and solution requirements 

and the information on a subsequent procedure is presented 
(JO-2 points ) 

■ Software enables learners to learn how to distinguish between various stimuli, objects, 
processes, terms, concepts (e.g. various species of plants and animals, chemical reactions, 
mental functions, etc.), provides learners with the distinguishment principles (or distinguishment 
signs), assigns tasks containing distinguishment and provides the correctness feedback 

(0—3 points) 

■ Software enables learners to learn how to generalize various stimuli, objects, processes, terms, 
concepts (e.g. learners search for common signs of particular object groups, create categories 
and subsequently define them; learners think about common signs of different situations and the 
method of responding to these situations, etc.), it can also provide the learners with the principles 
to be applied in generalization, assign the generalisation tasks, after each generalization provide 
the learners with the correctness feedback 

(0—3 points ) 

4.h Taking into account the requirements following from the cognitive architecture and its function 
(applying the cognitive learning theory) 

■ Simplicity, intuitiveness of learning activities in the software operation, 26 

■ When providing the information in electronic form, use a verbal and visual form thereof 
(0-2 points) 

■ Text (verbal) information and conversational information should be interconnected, as for time 
and space, with the picture information 

(0-2 points) 

■ Possibility for learners to enable and disable text or speech accompanying the pictures, 
animations, etc. 

(0-2 points) 

■ In the digital learning material (program) there should not be any redundant accompanying 
information, from the learning point of view 

(0-2 points) 

■ Respecting the perception principles (clear detachment of a figure - information of the 
substantial importance from the background - the accompanying information) using a shape, 
colour, size, intensity of stimuli, by means of schemes etc. 

(0-2 points) 

■ Presentations containing picture information should include verbal information provided in 
form of a speech and not a written text 

(0-2 points) 

■ Multimedia presentations in which visual materials are accompanied with speech, should not 
contain text legends or explanations 

(0-2 points) 

■ Ensure balance and functionality of static and dynamic elements (e.g. animations should be 
included only if they are functional - i.e. the objective is to provide demonstrative information 
of a procedural nature that contributes to the meaningful learning and not to distract attention 
towards efficient information, but of minor importance) 

(0-2 points) 

■ Provide learners with the preliminary information on the content and form of a multimedia 
presentation and its components as the precondition for production of simple schemes of 
acquired terms and concepts 

(0-2 points) 

■ Enable learners to select verbal or visual information in the multimedia presentation 
(0-2 points) 

■ Enable learners to select the presentation speed, interrupt the presentation, or repeat any of its 
components 

(0-2 points) 

■ Enable learners to select the content’s or a task’s difficulty level 
(0-2 points) 

■ Enable learners to select the methods and time of topic revision 
(0-2 points) 
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4.1 Taking into account individual differences in the level of general, academic intelligence 

■ Possibility to set several degrees of the curriculum content difficulty, reflecting various levels 
of cognitive functions of learners 

( 0-8 points ) 

■ Possibility to set the working speed, working time, content scope, and tasks for individual 
learners 

( 0-7 points) 

4.j Taking into account individual differences in learning and cognitive styles of learners 

■ Possibility to select visual or verbal modality of the provided information, 

■ When learning from the multimedia, enable the preference of visual or verbal modality of 
the information (or the possibility of more or less intensive highlighting or suppression of the 
information of certain modality) 

( 0-2 points) 

■ Representation of the sequence, hierarchy, structure of learning components and procedures 
that facilitate structuring of the curriculum content and learning among learners with analytical 
cognitive style 

( 0-2 points) 

■ Interconnection of learning components, curriculum segments, and concepts by means of 
hypertext links, facilitating thus the formation of more complex cognitive representations of the 
curriculum content among learners with holistic learning style 

( 0-2 points) 

■ Combination of procedures based on the programmed learning (learning material is structured 
into smaller segments gradually presented to a learner) with simulations, modelling, and 
experiments in the virtual environment (learners can use the experience-based learning, 
adjust parameters of graphical representation of a process or a phenomenon, and monitor the 
consequences of their own manipulations, test the hypotheses, make experiments, learn by 
exploration) 

( 0-2 points) 

■ Possibility to select a working speed, working time, content scope, type and difficulty level for 
individual learners 

(0-2 points) 

4.k Taking into account individual differences depending on the profile of multiple 
intelligences 

■ Content variety, diversity of learning activities of learners using their language skills and 
logical and mathematical skills, spatial imagination, physical skills, social abilities and skills, 
self-knowledge and self-regulation and occasionally also musical abilities and skills 

( 0-3 points) 

■ Assign tasks that stimulate learners to perform various activities (self-study, social interaction, 
performance of manual or other physical actions, etc.) 

(0-2 points) 

4.1 Taking into account individual differences following from different types ofpreferred learning 
motivation of learners 

■ For learners with prevailing performance motivation - provision for the feedback that enables 
learners to evaluate and compare their performance from different points of view, set the program 
parameters (various difficulty levels, various time intervals for the work with the program) 

(0-2 points) 

■ For learners with prevailing cognitive motivation - provision of hypertext links to extended the 
curriculum content, provision of various problems and creative tasks, solution of which satisfies 
the cognitive interests and enriches their knowledge and skills 

(0-2 points) 

■ For learners with prevailing social motivation - possibility within the program to cooperate, 
communicate with other persons (teacher, co-leamers, parents, experts, etc.), possibility for 
other persons to obtain the feedback on a learner’s results, as well as the possibility for learners 
to compare themselves with other learners in terms of the curriculum content acquisition 
(0-3 points) 
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10 

5 
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Altogether for area 4 
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Issamus elektroniniq mokymo priemoniij ir mokomosios 
programmes jrangos vertinimas 

Stefan KAROLCIK, Elena CIPKOVA, Roman HRUSECKY, Milan VESELSKY 

Nepaisant to, kad skaitmenines technologijos vis labiau naudojamos mokymo ir ugdymo pro- 
cese, vis dar truksta profesionalii} vertinimo priemoniij, kuriomis galima visapusiskai ir objektyviai 
nustatyti naudojamij skaitmeninip mokymo priemoniij kokyb^. Priemones „Issamus elektroninip 
mokymo priemoniij ir mokomosios programmes jrangos vertinimas" (CEELTES) konstravimas 
buvo paremtas keletu tyrimij ir ziniomis is skaitmeninip mokymo ir mokymosi priemoniij kurimo 
bei jp jgyvendinimo mokymo procese. Vertinimo priemone susideda is kriterijp rinkinip (kata- 
logp), kuriuos sudaro 4 atskirai vertinamos sritys: techninip, technologinip ir vartotojp atributp 
sritis; turinio, operacijp, infonnacijos strukturizavimo ir apdorojimo sritis; informacijos apdoro- 
jimo mokymo, atpazinimo ir svietimo poziuriu sritis; psichologinip ir pedagoginip skaitmeninip 
produktp sritis. Nurodytas sritis nepriklausomai vertino atitinkamos mokslines srities specialistai. 
Galutinis sip skaitmeninip produktp vertinimas leidzia nuspr^sti, ar tam tikra skaitmenine moky¬ 
mo priemone gali buti jtraukta j mokymo procesp. 



